Courtroom Applications of Virtual Environments, Immersive Virtual Environments, and Collaborative Virtual Environments

This paper examines the possibilities and implications of employing virtual environments (VEs), immersive virtual environments (IVEs), and collaborative virtual environments (CVEs) in the courtroom. We argue that the immersive and interactive reality created by these tools adds significant value as a simulation of experience to enhance courtroom practice. The obvious boundaries between real and virtual enhance the attractiveness of these tools as technologies of rhetorical persuasion that can be used to demonstrate subjective perspective, strengthen or impeach the credibility of witnesses and provide the trier of fact with a better understanding of each side’s perception of the facts at issue. In the first section, we introduce the concepts of VEs, IVEs, and CVEs, describe the manners in which these technologies have been applied to settings other than the courts system, and review the relevant psychological and legal literature. In the second section, we discuss specific applications of the technology to the court system and suggest how it could improve upon current procedures. Finally, in the third section we discuss some of the limitations and problems. This last section suggests legal reforms necessary to the adoption of these technologies, specifically rules of procedure that provide for all parties to be able to access, manipulate and inspect any virtual environment, the trier of fact to be able to interact with, rather than just accept the lawyer’s rendition, and rules that provide for the parties to introduce at trial an inventory of all digital assets contained in the virtual environment, making those that are stipulated to and those that are in controversy. IVEs and the Courtroom 3 While virtual environment technology is not yet fully realized, eventually it will provide distinct advantages to litigators within and outside of the courtroom. Its adoption should be promoted and even underwritten by the courts because this technology offers practical advantages for recreating crime and accident scenes, preparing witnesses, and experts, and conducting police lineups. But these immersive virtual environments are more than just faster videoconferencing techniques. While they offer practical advantages, as we shall discuss, they also represent a qualitative advance over earlier technologies. Unlike prior tools used for recreations and simulations, these are both immersive and interactive. There are those that argue that the risk of manipulation should prevent multimedia from being adopted in trial practice, but it is precisely because these new technologies produce simulated yet interactive reality that they are an ideal technique for rhetorical persuasion and argument. They are particularly well-suited for use in contexts where a subjective measurement of perspective is called for and where that perspective needs to be tested and even impeached. This article seeks to provide the raw material – an understanding of the technology – to argue for the use of these technologies and to enable lawyers and policymakers to make informed decisions about how these technologies will be introduced into the legal process. The immersive, interactive and highly mutable quality of these virtual environments does not vitiate their value to the legal process. Rather the characteristics of the technology point toward adopting procedural rules that allow all parties to “play” with the virtual reality simulations. This means that judge, jury and litigants should be able to test the immersive experience. All parties should have access to the simulation to be able to alter IVEs and the Courtroom 4 the perspective and impeach the credibility of the simulation. Finally, parties should be required to submit a list of “assets” or virtual objects included in a simulation and to mark graphically within the simulation those that are stipulated those, those that constitute dramatic interpretation and those that are known to be controverted. Introduction to Virtual Environments The term virtual reality has been widely used and often creatively exaggerated by Hollywood producers and science fiction writers for decades. Consequently, there are many misconceptions and expectations about the nature of the technology. For the purposes of this paper, we define virtual environments (VEs) as “synthetic sensory information that leads to perceptions of environments and their contents as if they were not synthetic” (Blascovich, et al., 2002, pg. 105). Typically, digital computers are used to generate these images and to enable real-time interaction between a user and the VE. In principle, people can interact with a VE by using any perceptual channel, including visual (by wearing a head-mounted display with digital displays that project objects in the VE), auditory (by wearing earphones that are conducive towards playing sounds that seem to emanate from a specific point in space in the VE), haptic (by wearing gloves that use mechanical feedback or air blasts towards the hands when a person makes contact with an object in the VE), olfactory (by wearing a nosepiece that releases different smells when a person approaches different objects in a VE), or gustatory senses. Our definition of VE would include non-digital information. For instance, a scarecrow in a field is an example of physical, synthetic, sensory misinformation that deceives crows into thinking the farmer is guarding the crops. Along similar lines, IVEs and the Courtroom 5 lawyers have often employed physical virtual environments in courtrooms, for example using physical objects to indicate a suspect’s and witnesses’ relative positions. Like the scarecrow that is meant to deceive the crow into thinking that the farmer is in the garden, VE technologies create a richly-instantiated but still simulated version of reality. The boundaries of the physical virtual environment are evident, allowing the use by lawyers in the courtroom of these tools as a mechanism for argumentation and persuasion. In this report, however, we focus on digital VEs. Current software (i.e., Vizard, 3D Studio Max, 3dMeNow, Poser) makes it quite easy to produce digital virtual worlds, and consequently, digital VE simulations can be produced to fit almost any specific application with only moderate degrees of cost and effort. The similarity between nondigital and digital VEs remains important; researchers and litigators who employ digital VEs are not engaging in any qualitatively new or unsound technique. Instead of using maps, charts and cardboard cutouts, proponents of VEs are using computers. An immersive virtual environment (IVE) is one that perceptually surrounds the user of the system. Consider a child’s video game; playing that game using a joystick and a television set is a VE. On the other hand, if the child were to have special equipment that allowed her to take on the actual point of view of the main character of the video game, that is, to control that character’s movements with her own movements such that the child is actually inside the video game, then she is interacting in an IVE. In other words, in an IVE, the sensory information of the VE is more psychologically prominent than the sensory information of the physical world. For this to occur, IVEs often employ two characteristic features. First, the users are unobtrusively tracked as they interact with the IVE. User actions such as head orientation and body IVEs and the Courtroom 6 position are automatically and continually recorded and the IVE in turn is updated to reflect the changes resulting from these actions. In this way, as a person in the IVE moves, the tracking technology senses this movement and renders the virtual scene to match the user’s movement. Second, sensory information from the physical world is kept to a minimum. For example, in an IVE that relies on visual images, the user wears a head-mounted display (HMD) or sits in a dedicated projection room. By doing so, the user cannot see the objects from the physical world, and consequently it is easier for them to become enveloped by the synthetic information. Two important features of IVEs that will continually surface in later discussions are: 1) IVEs necessarily track a user’s position and head direction, facial expression and sometimes eye direction to render the scene, providing a wealth of information about where the user is focusing his or her attention and what can be observed from that specific vantage point, and 2) The designer of an IVE has a tremendous amount of control over the user’s experience, and can design the virtual world to look and feel in almost any desired manner. IVEs and the Courtroom 7 IVEs can be configured and displayed in a number of ways. Figure 1 shows one Figure 1: A depiction of an HMD based IVE. The components are: 1) position tracking cameras. 2) HMD and orientation tracking sensor, and 3) image generator. example, a system in which a user wears an HMD that displays images via small computer screens over each eye. As the user in this example interacts with the virtual world, an optical motion sensor tracks his position in the room and an inertial motion sensor tracks his head orientations. Both of these devices are lightweight enough to be attached to the user's HMD and provide accurate and fast screen updates, that is, a constant redrawing of the VE for the user as a direct function of his translation and orientation. In other words, sixty times per second, the system redraws the objects in the virtual world. Every time it redraws the virtual world, it checks to see if the user, in the physical world, has turned his or her head, or if the user has walked in any direction. If he or she has changed their orientation or position, then the system redraws the virtual objects in the VE to reflect those exact changes. In this sense, the person moves through IVEs and the Courtroom 8 the VE in the same way that he or she would move through the physical world. Furthermore, the HMD provides the user with distinct images for each eye, providing stereoscopic depth cues, the information concerning distance and depth that people receive from having two eyes facing in the same direction, inside the IVE. Together, this system convinces the user's perceptual system that he or she is contained inside an actual 3D world and allows the user to actively explore that world in any manner he or she chooses. Research demonstrates that people walking through IVEs can navigate and perceive directional information quite proficiently, almost as well as in the physical world (Chance, Guanet, Beall, & Loomis, 1998), although operating joystick-based desktop VEs causes spatial performance to drop (Richardson, Montello, & Hegarty, 1999). An alternative IVE configuration is a Cave Automatic Virtual Environment (CAVE). In this system, the user stands inside a cube-shaped room with rear-projection screens as walls. The user’s position is usually tracked by a type of an electromagnetic device and orientation tracking is unnecessary (since the world is projected all around the user on the six sides of the cube). However, instead of wearing the HMD, the user wears shutter-glasses (for stereoscopic vision) and receives updated visual images by looking at the screens covering the walls. A large amount of research regarding IVEs centers on the notion of presence, the degree to which the user actually feels as if they are present in the IVE (as opposed to present in the physical world). A wealth of research seeks to understand the phenomenon of presence: understanding the mechanisms that underlie the subjective experience of “being in another world” strikes at the very heart of the virtual reality experience. To IVEs and the Courtroom 9 validate IVEs as a usable courtroom technology, it is important to consider the extent to which a user is immersed in the digital world created by the lawyer (as opposed to the physical courtroom). Attempts at capturing the subjective experience of presence in an objective manner have proceeded along several different lines, including questionnaire ratings (Short, Williams, & Christie, 1976; Held & Durlach, 1992; Witmer & Singer, 1998), physiological measures (Meehan, 2001; Weiderhold, Gervirtz, &, Wiederhold, 1998), and behavioral measures (Zahorik, 1998; Meehan, 2001; Bailenson, Beall, & Blascovich, 2002; Mania & Chalmers, 2001; Welch, 1999). Despite broad research on the topic of presence, reliable, objective measures are still lacking, and much debate as to how to improve current measures continues (Loomis, 1992; Lombard & Ditton, 1997; Slater, 1999). Often times, multiple people interact with one another inside of the same IVE or VE; this arena is called a Collaborative Virtual Environment (CVE). A basic example of a CVE is an Internet chat room. In these collaborative environments, users may or may not be located in the same physical environment; however, their movements, nonverbal behaviors, and voices are all projected into a single VE or IVE. In CVEs, a user is typically represented by some type of a visual form called an avatar (see Bailenson & Blascovich, 2002, for an extended definition). In addition to experiencing presence in a CVE, users also experience social presence (also known as copresence) while interacting with other avatars. Social presence reflects the degree to which a user of a CVE feels that he or she is in the presence of and interacting with other veritable human beings (Blascovich et al. 2002; Bailenson, Blascovich, Beall, & Loomis, 2002, 2001; Heeter, IVEs and the Courtroom 1